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1  Introduction 


In  the  design  of  many  fault-tolerant  distributed  protocols,  processors  are  as¬ 
sumed  to  be  the  only  faulty  components;  communication  channels  between 
processors  are  assumed  to  be  failure-free.  This  is  particularly  evident  in 
the  large  amount  of  literature  devoted  to  distributed  agreement  protocols 
Fis83  .  This  unequal  treatment  of  processor  and  communication  failures 
has  been  justified  on  the  grounds  that  any  failures  a  channel  exhibits  can  be 
attributed  to  either  one  of  the  two  processors  the  channel  connects.  This 
way  of  modeling  channel  failures  is  not  satisfactory.  Hadzilacos  Had86 
wrote: 

...  we  prefer  to  consider  a  component  faulty  only  if  it  misbe¬ 
haves,  not  if  other  components  related  to  it  misbehave.  More¬ 
over,  in  BA  [Byzantine  Agreement],  pronouncing  a  processor 
correct  or  faulty  is  not  merely  a  question  of  accounting  for 
faults:  a  processor  that,  ...  ,  is  faulty  is  exempted  from  the 
requirement  to  decide  on  a  value  subject  to  the  Agreement  and 
Validity  conditions. 

In  this  paper,  we  study  the  problem  of  designing  protocols  that  toler¬ 
ate  failures  of  communication  channels.  We  consider  those  protocols  that 
guarantee  knowledge  gain  between  processors  via  message  transfers.  (We 
call  these  non-trivial  protocols.)  We  show  that  for  asynchronous  systems, 
non-trivial  protocols  cannot  be  guaranteed  to  terminate  even  if  we  assume 
that  only  transient  communication  failures  can  occur,  and  we  only  want 
to  achieve  a  weak  kind  of  termination.  The  same  result  also  holds  for 
synchronous  systems. 

Informally,  a  protocol  weakly  terminates  if  from  every  point  of  its  execu¬ 
tion.  the  execution  can  be  continued  to  a  point  at  which  all  processors  stop. 
Channel  failures  are  transient  if  any  message  sent  repeatedly  is  eventually 
received.  Transient  channel  failure  is  the  model  usually  adopted  by  proto¬ 
col  designers  to  account  for  the  faulty  behavior  of  communication  Links  in 
long-haul  and  local  computer  networks,  e.g.,  ARPANET  and  Ethernet. 

Several  previous  results  are  related  to  the  problem  of  termination  in  the 
presence  of  transient  channel  failures.  With  permanent  channels  failures,  it 
is  well-known  that  it  is  impossible  to  achieve  common  knowledge  (or  "co- 


ordinated  attack”  (Gra78j),  or  even  eventual  common  knowledge  HM85b  . 
In  contrast  to  these  r  suits,  we  consider  transient  channel  failures,  we  do 
not  restrict  our  study  to  protocols  that  achieve  either  common  knowledge 
or  eventual  common  knowledge,  and  we  concentrate  on  the  problem  of  ter¬ 
mination.  These  differences  are  underscored  by  the  fact  that  with  transient 
channel  failures,  there  is  a  protocol  for  achieving  eventual  common  knowl¬ 
edge.  However,  this  protocol  is  non- terminating.  (See  Appendix  A.) 

In  his  study  of  commit  protocols,  Skeen  SkeS2  showed  that  processors 
may  have  to  block  forever,  neither  committing  or  aborting.  ( i . e . .  the  commit 
protocol  may  not  terminate),  if  the  communication  network  is  permanently 
partitioned.  In  contrast  to  this  result,  we  show  that  even  if  only  transient 
channel  failures  may  occur  (and  hence,  even  if  the  network  is  not  partitioned 
permanently),  any  non-trivial  protocol  (including  commit  protocols)  can¬ 
not  be  guaranteed  to  terminate.  Therefore,  our  result  complements  and  or 
generalizes  previous  impossibility  results  Gra78,H\I85b,Ske82  . 

In  this  paper,  we  prove  our  result  onh  for  asynchronous  systems.  A 
similar  proof  can  show  that  it  also  holds  for  synchronous  systems.  The 
paper  is  organized  as  follows:  a  formal  model  of  asynchronous  system  is 
in  Section  2.  We  present  the  result  in  Section  3.  Section  4  contains  the 
discussion. 

2  Model  of  an  asynchronous  distributed  system 

A  distributed  system  consists  of  m  processors  that  communicate  by  mes¬ 
sages  via  communication  channels.  Each  processor  is  a  deterministic  state 
machine,  which  may  have  an  infinite  number  of  states.  In  each  state,  a 
processor  can  execute  zero  or  more  atomic  actions.  The  states  in  which 
a  processor  cannot  execute  any  atomic  actions  are  called  terminal  states. 
Executions  of  atomic  actions  are  called  events.  Any  event  may  cause  a 
processor  to  change  its  state.  Two  possible  events  of  a  processor  i  are: 

1.  send, (processor,  message),  which  i  executes  to  send  message  to  an 
other  processor:  and 

2.  receive, (processor,  message ),  which  t  executes  to  receive  message  from 
another  processor. 


2 


2.1  Runs  of  processors 


Let  state,, q  be  an  initial  state  of  processor  t,  and  s ,  be  a  sequence  of  events 
of  i.  The  pair  h,  =  [state, s,)  is  a  local  history  of  i  if  s,  is  a  sequence 
of  events  that  i  executes  beginning  at  state  state, ,Q.  Let  ht. state  denote 
state,  o,  and  h,. events  denote  s,.  A  local  history  h ,  is  finite,  if  h,. events  is 
finite. 

A  m-tuple  of  local  histories,  hist  =  {hi ,  h2, .  . . ,  hm},  is  a  system  history 
if 

1.  7i  :  1  <  i  <  m,  h,  is  a  local  history  of  processor  i:  and 

2.  h,. events  contains  receive,^',  msg),  only  if  hr  events  contains 
send^i,  msg). 

If  all  hfi s  are  finite  local  histories,  hist  is  a  finite  system  history.  Since 
all  processors  are  deterministic,  any  finite  system  history  unambiguously 
specifies  the  state  of  each  processor. 

An  asynchronous  run  r  is  a  pair  {hist,  mesg)  such  that  hist  is  a  finite 
system  history,  and  mesg  is  a  subset  of  the  set  of  messages  that  are  not 
yet  delivered;  i.e.,  those  messages  that  are  sent  and  not  received  in  hist.1 
The  two  components  of  r  are  denoted  by  r.hist  and  r.mesg.  respectively. 
Messages  that  are  sent  and  not  received  in  hist ,  and  also  are  not  in  mesg 
are  lost. 

To  model  message  losses,  vve  introduce  the  following  notation:  r'  - 
failure{r ),  if  runs  r'  and  r  are  identical  except  that  some  messages  that 
are  not  yet  delivered  in  r  are  lost  in  r' .  To  be  more  precise,  let  r  =  [hist, 
mesg)  and  r'  —  ( hist '.  mesg').  If  hist'  -  hist  and  mesg'  I  mesg.  then  r'  — 
failure(r).  (See  Figure  1.) 

An  asynchronous  system  is  defined  as  the  set  of  all  asynchronous  runs. 
Each  run  corresponds  to  a  possible  state  of  the  system.  Changes  of  system 
states  are  modeled  by  continuations  of  runs.  A  run  r'  is  a  continuation  of 
a  run  r,  if  r'  and  r  meet  the  following  conditions;  Let  h,  be  proj,{  r.hist), 
and  h[  be  proj,(r'  .hist). 

lA  lock-step  synchronous  run  {hut,  mesg)  DDS87  must  meet  the  following  additional 
requirements  For  all  processors  i  and  j,  the  numbers  of  events  that  i  and  j  have  respec¬ 
tively  in  hist  differ  by  at  most  one;  and  for  all  msg  ~  mesg,  if  the  sender  of  msg  is  t.  then 
the  sending  of  msg  is  t’s  last  event  in  hist 
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1.  V*  :  1  <  i  <  m,  h\. state  =  hx. state,  and  hx. events  is  a  prefix  of 
h\.  events; 

2.  for  all  receive^',  msg )  that  are  in  r'  and  not  in  r,  either  msg  6  r.mesg, 
or  the  event  send }(i,msg)  is  in  r'  but  not  in  r;  and 

3.  for  all  messages  msg  €  r'.mesg,  either  msg  £  r.mesg,  or  the  sending 
of  msg  is  an  event  in  r'  but  not  in  r. 

We  say  r  =  prefix(r'),  if  and  only  if  r'  is  a  continuation  of  r. 

We  now  describe  the  channel  failure  model.  We  assume  that  all  channels 
exhibit  only  transient  omission  failures.  If  copies  of  a  message  are  sent 
repeatedly  over  a  channel,  at  least  one  copy  will  eventually  be  received. 
However,  there  is  no  bound  on  the  number  of  messages  that  may  be  lost. 
To  simplify  our  discussion  of  lost  messages,  we  assume  that  every  copy  of  a 
message  is  unique.  From  this  assumption  and  the  definitions  of  runs  and  of 
continuations,  a  message  lost  in  a  run  is  also  lost  in  all  of  its  continuations. 

Events  are  partially  ordered  by  the  after  relation  Lam78  .  Event  e'  is 
after  event  e  in  run  r  if  and  only  if 

1.  e  and  e'  are  events  of  the  same  processor,  and  a  prefix  of  r  contains 
e  but  not  e'; 

2.  e  is  the  sending  of  a  message  msg  and  e'  is  the  receipt  of  msg;  or 

3.  there  is  an  event  e"  such  that  e'  is  after  e",  and  e"  is  after  e. 

If  e'  is  after  e,  then  we  say  that  e  is  before  e' . 


2.2  Distributed  Protocols 

A  local  protocol  P,  of  a  processor  i  is  a  function  from  fs  current  state  and 
the  sequence  of  messages  that  i  has  received  to  the  next  atomic  action  to  be 
executed  by  i.  (Our  results  can  be  generalized  to  allow  non-deterministic 
protocols.)  A  distributed  protocol  is  a  n-tuple  P  =  { Pi ,  P2 .  ■  ■  ■ ,  Pn}  such 
that  for  all  1  <  1  <  n,  P,  is  a  local  protocol  of  processor  i.  A  system 
history  {h\,  •  ■  •  ,hn}  is  consistent  with  P  if  for  all  1  <  1  £  n,  the  sequence 
of  events  in  hx  corresponds  to  an  execution  of  local  protocol  P,  beginning 
from  the  initial  state  of  h,.  A  run  r  is  consistent  with  protocol  P  if  r.hist  is 


consistent  with  P.  If  r  is  consistent  with  P,  so  do  prefix (r )  and  failure(r). 
For  convenience,  protocol  P  can  be  identified  with  the  set  of  asynchronous 
runs  that  are  consistent  with  it. 

The  termination  property  of  a  protocol  P  is  characterized  by  the  runs  in 
P .  A  terminating  run  is  one  in  which  all  processors  enter  terminal  states.  A 
weakly  terminating  run  is  one  that  has  at  least  one  terminating  continuation 
in  P.  Runs  that  are  not  weakly  terminating  are  non-terminating .  Examples 
of  non-terminating  runs  are  runs  in  which  the  system  is  deadlocked,  and 
runs  in  which  one  processor  is  in  an  infinite  loop.  Protocol  P  is  weakly 
terminating ,  if  all  its  runs  are  weakly  terminating;  it  is  non-terminating 
otherwise. 

3  Problem  of  Protocol  Termination 

In  this  section,  we  prove  that  any  protocol  that  guarantees  knowledge  trans¬ 
fer  despite  transient  communication  failures  is  non-terminating.  Roughly, 
our  argument  goes  as  follows.  First,  given  any  weakly  terminating  pro¬ 
tocol  and  any  initial  system  state,  we  show  that  starting  from  this  state, 
this  protocol  must  have  a  run  that  terminates  without  message  transfers. 
Then,  we  note  that  without  message  transfers,  knowledge  cannot  be  trans¬ 
ferred  in  asynchronous  systems  CM86;.  Hence,  in  asynchronous  systems 
with  transient  communication  failures,  weakly  terminating  protocols  can¬ 
not  guarantee  knowledge  transfer. 

3.1  Termination  without  Message  Exchange 

An  event  is  a  last  receive  event  in  a  run  if  it  is  not  before  another  receive 
event  in  this  run.  A  run  r  is  an  initial  run  if  its  contains  no  events:  it 
includes  only  the  initial  states  of  the  processors. 

Theorem  1  Ltt  P  he  a  weakly  terminating  protocol.  For  all  initial  runs  r 
in  P ,  r  has  a  terminating  continuation  in  P  in  which  no  processor  recti  res 
any  messages. 

Proof:  By  contradiction.  Let  r  be  an  initial  run  in  P  such  that  even- 

terminating  continuation  of  r  in  P  contains  at  least  n  (n  >0)  receive 
events.  Let  r'  —  (hist1,  mesg')  be  a  terminating  continuation  of  r  in  P  rhat 


contains  exactly  n  receive  events.  Let  e  be  a  last  receive  event  in  r',  and 
suppose  that  e  occurs  at  processor  i. 

Delete  e  and  all  events  that  are  after  e  from  hist'.  It  is  easy  to  see  that 
this  results  in  a  system  history  hist"  (See  Figure  2).  Let  run  r"  =  ( hist ", 
d>).  Note  that  r"  =  failure ( prefix (r')).  Since  r'  satisfies  P.  r"  must  also 
satisfy  P;  thus,  r"  is  also  in  P.  Moreover,  r"  is  a  run  that  has  only  n  —  1 
receive  events. 

Since  e  is  a  last  receive  event  of  r\  all  events  that  are  deleted  from  r' 
in  the  construction  of  r"  occur  only  at  processor  i.  The  histories  of  all 
processors  except  i  in  r'  and  r"  are  the  same.  Since  r'  is  a  terminating  run. 
i  is  the  only  processor  in  r"  that  has  not  terminated.  By  construction,  the 
channels  of  r"  are  empty.  Hence,  in  any  continuation  of  r",  i  receives  the 
same  number  of  messages  as  it  does  in  r",  namely  n  —  1.  Since  P  is  weakly 
terminating,  r"  has  a  terminating  continuation  r*  in  P.  However,  run  r* 
contradicts  the  minimality  of  n.  I 

3.2  Processors’  Knowledge 

3.2.1  Syntax 

We  adopt  the  notation  used  by  Halpern  and  Moses  HM83a  to  describe 
the  knowledge  of  processors.  Let  ^  be  a  set  of  primitive  propositions 
{pj.p2.  .  .  .}.  The  language  £($)  is  the  smallest  set  of  formulas  containing 
closed  under  /'  .  and  modal  operators  A’[,  A’2, ....  and  A’m.  Formulas 
of  the  form  p  .  q  are  abbreviations  for  ~(~p  ~q).  p  Z  q  are  for  -ip  ’  ~q). 

3.2.2  Semantics 

For  all  processors  li  :  1  <  i  <  m ,  i’s  view  of  a  run  r  is  the  projection  of  r .hist 
pro]t(  r.hist),  on  t.  The  views  of  processor  i  divide  runs  of  a  protocol  into 
equivalence  classes.  Runs  r  and  r’  are  in  the  same  equivalence  class  (with 
respect  to  i ),  if  and  only  if  projfir.hist)  =  proj,(r' .hist).  The  equivalence 
class  of  r  according  to  i,  denoted  by  possfir).  determines  what  i  can  know 
at  the  end  of  r. 

A  processor's  knowledge  at  the  end  of  a  run  is  defined  inductively.  Let 
P  be  a  protocol,  and  r  be  a  function  mapping  from  to  the  set  of  subsets 


of  runs  of  P.  Let  r  f?  denote  that  formula  p  is  interpreted  to  be  true  in 
r.  If  p  is  a  primitive  proposition  (p  £  3>), 

r  =  p  iff  r  6  7r(p). 

For  formulas  ~p  and  p  A  q  where  p,q  c  £($), 

r  t=  ~p  iff  not  r  =  p; 
r  =  p  A  q  iff  r  =  p  and  r  =  q. 

And  finally,  processor  i  knows  p  in  r,  if  and  only  if  p  is  true  in  all  the  runs 
that  are  equivalent  to  r.  Ip  6  £($), 

r  j=  A',p  iff  Vr'  £  posst(r),r'  j=  p. 

A  protocol  P  is  said  to  guarantee  a  formula  p,  if  every  run  r  of  P  has  a 
continuation  r'  in  P  such  that  r'  j=  p. 

3.3  Termination  without  Knowledge  Transfer 

In  this  section,  we  show  that  in  the  presence  of  transient  communication 
failures,  a  weakly  terminating  protocol  cannot  guarantee  the  transfer  of  a 
processor's  local  knowledge  to  another  processor.  In  other  words,  protocols 
that  guarantee  knowledge  transfer  are  necessarily  non-terminating. 

Chandy  and  Misra  CM86’  defined  that  a  formula  p  is  local  to  a  processor 
i  with  respect  to  protocol  P,  if  Vr  c.  p,  r  =  h\p  J  h\~p.  They  show  that 
in  an  asynchronous  system,  a  processor  must  receive  messages  to  acquire 
knowledge  of  a  non-local  formula: 

Lemma  1  (CM86)  Let  p  be  a  formula  that  is  not  local  to  processor  i  and 
is  local  to  some  other  processors  in  protocol  P.  [f  for  some  run  r  and 
its  continuation  r'  in  P,  r  =  -(A",  /  A’,-p)  and  r'  =  h\p  -  A’,-p,  then  i 
receives  at  least  one  more  message  in  r'  than  it  does  in  r.  — 

A  formula  p  is  an  a  priori  formula  of  i  in  protocol  P,  if  for  all  initial 
runs  r  of  P,  r  =  A’,p  ,  Kt-p.  Obviously,  if  p  is  not  a  priori  to  i.  then  it  is 
not  local  to  i. 

Theorem  2  Let  P  be  a  protocol ,  and  p  c  £($)  a  formula  that  is  not 
a  priori  to  processor  i  and  is  local  to  some  other  processors  m  P.  If  P  is 
weakly  terminating,  it  cannot  guarantee  h\p  v  A’,  -p. 


Proof:  By  contradiction.  Without  loss  of  generality,  suppose  that  P 

guarantees  K,p  \  K,-p  and  is  weakly  terminating.  Since  p  is  not  a  priori  to 
i,  P  has  an  initial  run  r  such  that  r  =  -(A',p  v  h\~p).  Since  P  is  weakly 
terminating,  by  Theorem  1,  r  has  a  terminating  continuation  r'  in  P  in 
which  no  messages  are  received.  Furthermore,  since  r  =  -(A \p  V  K,~p) 
and  no  messages  are  received  in  r',  by  Lemma  1,  r'  =  -(A',p  v  A',-p)  . 
Thus  P  does  not  guarantee  K,p  ,  A', ~p.  a  contradiction.  - 


4  Discussion 

We  have  showed  that  in  asynchronous  systems  with  transient  channel  fail¬ 
ures.  only  non-terminating  protocols  can  guarantee  transfer  of  knowledge. 
This  result  can  be  extended  to  systems  with  synchronous  processors  and 
synchronous  communication. 

Synchronicitv  is  a  critical  parameter  of  the  problem  of  reaching  agree¬ 
ment  in  the  presence  of  processor  failures.  In  asynchronous  systems,  there 
are  no  deterministic  solutions  even  if  only  one  processor  may  fail  by  halt¬ 
ing  FLP85  ;  in  synchronous  systems,  however,  several  solutions  are  known 
DDS8T,Fis83'.  In  contrast,  our  negative  result  holds  for  both  synchronous 
and  asynchronous  systems. 

Since  weakly  terminating  protocols  do  not  guarantee  knowledge  trans¬ 
fer.  we  may  have  to  settle  for  protocols  that  guarantee  only  that  all  but  one 
processors  will  terminate.  The  following  problem  serves  as  an  illustration. 
Processors  i  and  j  are  connected  by  a  link  with  transient  failures.  We  want 
a  protocol  that  i  can  use  to  send  a  message  m  to  j  such  that 

1.  j  receives  m  from  i,  and 

2.  i  and  j  are  eventually  allowed  to  forget  m. 

Such  a  protocol  P  is  given  in  Figure  3  Ske86  .  Since  channel  failures  are 
transient,  one  of  the  copies  of  m  that  i  sends  repeatedly  to  j  is  guaranteed 
to  be  received  by  j.  Similarly,  the  acknowledgements  ack(m)  from  j.  and 
ack(ack(m ))  from  i,  will  also  be  received  by  i  and  ;,  respectively.  It  is 
now  easy  to  see  that  protocol  P  achieves  the  two  goals.  Note,  however, 
that  processor  i  never  terminates  (it  will  remain  in  the  do-forever  loop). 
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Let  ack  denote  acknowledgements. 


processor  i 

repeat  until  receipt  of  ack(m) 
send  m  to  j; 

od; 

forget  m: 
do  forever 

if  ack(m)  is  received  then 
send  ack(  ack(  m))  to  j: 

od. 


processor  j 
wait  to  receive  m; 

repeat  until  receipt  of ■  ack  (ack  (m)) 
send  ack(m)  to  i ; 

od; 

forget  m; 

stop. 


Figure  3:  Protocol  P. 


Our  result  shows  that  this  is  not  a  deficiency  of  this  particular  protocol:  in 
a  system  with  transient  channel  failures,  any  protocol  that  achieves  goals 
(1)  and  (2)  also  guarantees  that  j  knows  m;  therefore,  it  must  be  non¬ 
terminating. 
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Let  ack°(m)  denote  m,  and  acA:*-1^)  denote  the  acknowledgement  to  ackk(m). 


processor  i 


k  :  =  0; 

do  forever 

repeat  until  receipt  of  ack‘t~l(m) 
send  ackk(m)  to  j; 

od; 

k  :=  k  -  2; 

od. 


srocessor 


k  :=  1; 
do  forever 

wait  to  receive  ackk~1(m ); 
repeat  until  receipt  of  ackk~l(m) 
send  ackk(m)  to  i: 

od: 

k  :  =  k  —  2: 

od. 


Figure  4:  Protocol  Q  allows  processors  i  and  j  to  attain  eventual  common 
knowledge  of  m. 

Appendix  A:  Achieving  Eventual  Common  Knowledge 

We  present  a  protocol  that  allows  processors  i  and  j  to  gain  eventual  com¬ 
mon  knowledge  of  a  fact  m  despite  transient  channel  failures.  We  give  only 
an  informal  description  here.  (The  formal  definition  of  eventual  common 
knowledge  is  in  HM85b  .) 

A  fact  is  eventually  true  in  a  run  r.  if  it  must  become  true  some  time 
in  the  “future'1  of  r.  Furthermore,  a  fact  m  is  stable  with  respect  to  a 
protocol  P,  if  for  all  run  r  ~  P.  once  m  becomes  true  in  a  run  r,  it  remains 
true  in  all  continuations  of  r  in  P.  Eventual  common  knowledge  of  a  stable 
fact  m  is  achieved  in  a  run  r,  if  in  r,  m  is  true,  and  that  eventually  every 
processor  knows  that  m  is  true,  and  that  eventually  every  processor  knows 
that  eventually  every  processor  knows  that  m  is  true.  •  •  ■ ,  ad  infinitum. 

Let  m  be  a  stable  fact  and  suppose  that  processor  i  knows  that  m  is 
true.  We  claim  that  the  non-terminating  protocol  Q  in  Figure  4  allows 
processors  i  and  j  to  attain  eventual  common  knowledge  of  m,  despite 
transient  channel  failures.  It  is  clear  that  the  repeated  sending  of  ack°(m) 
by  processor  i  guarantees  that  at  least  one  a ck°(m)  will  arrive  at  processor 
j.  Hence,  j  will  send  acki(m)  to  i  repeatedly  until  at  least  one  acAi^m) 
arrives  at  i.  Thus,  it  follows  by  induction  that  for  all  k  >  0,  at  least 
one  a ck2k(m)  will  arrive  at  j,  and  at  least  one  ack2k~l{m )  will  arrive  at  i 


despite  transient  channel  failures.  Let  E'm  denote  that  both  i  and  j  know 
m,  and  Ek  1m  denote  that  both  i  and  j  know  Ekm.  It  is  easy  to  see  that 
the  receiving  of  ac*°(m)  by  j  implies  E'm.  In  general,  for  all  k  >  0  the 
receiving  of  ack^(m)  by  j  implies  E^'m.  Thus,  despite  transient  channel 
failures,  i  and  j  achieve  eventual  common  knowledge  of  m  by  executing 
protocol  Q.  Note,  however,  that  neither  t  nor  j  ever  stop  executing  Q 


